What does this PR do ?

CPU-only unit tests for one property of #3480:

Pausing a run to checkpoint and resuming it should train on exactly the same prompt groups as never pausing at all.

Draft, targeted at amahishi/sc-tq-native-recovery rather than main, so the tests land with the feature. Tests only — no source changes.

Result

10 failed, 19 passed, 4 xfailed in 11.69s      (CPU only, no GPU)

Visual walkthrough of all six cases and all eighteen outcomes:
https://terrykong.github.io/gh-pages-poc/terryk/pr-3827-test-matrix.html

The 10 failures are deliberate and are the point of this PR. They are not "tests I could not get working" — they are groups that came back before this change and do not now.

Three files, split by which bar a case misses

There are two bars, and conflating them hides the signal:

A step backwards should be loud, so the regression file has no xfail marks. A gap that was always there is a missing feature, so those are xfail(strict=True) — a fix turns them into XPASS failures and the row gets moved into PASSING.

What the previous behaviour actually was

Checked against the merge base rather than assumed:

• the buffer was saved on every checkpoint, no capability gate — single_controller.py:802
• the restore ran for any sampler, gated only on the saved sampler name matching — :270 — and both sides read self._async_cfg.sampler.name, so it always matches on a same-sampler resume
• state_dict saved the committed slots and dropped in-flight ones, in as many words: "Unready reservations are in-flight rollouts and are dropped, matching legacy semantics."

So every fully generated group used to survive a restart under every sampler, and in-flight loss is pre-existing. That is exactly the line between the two files.

The regressions

in_order and weight_fifo declare supports_buffer_checkpoint = False, so no sidecar is written (:981) and the restore returns early (:322):

AssertionError: in_order/lag1-next-step-complete: ['g12', 'g13', 'g14'] came back
before this change and do not now. These groups finished generating and committed,
so the old save wrote them and the old restore returned them on a same-sampler
resume. recovered=[]

Two gated rows pass, which is the assertion working rather than blanket-failing: in trained-what-was-ready-leaving-a-hole the only untrained groups are in-flight, so the old code recovered nothing either and there is nothing to regress.

test_the_rows_are_still_in_transfer_queue_after_a_gated_restore passes too — TransferQueue still holds every committed row. The regression is the index, not the tensors, so a fix can be index-only.

The pre-existing gap

A group that has not committed is never saved: reserve() marks the slot not-ready (replay_buffer.py:910) and metadata_state_dict skips it (:1093). A group is reserved or committed with nothing in between, so "one rollout still running" and "not started" are the same thing to a checkpoint — group 13 in S_PARTIAL finished both rollouts and is dropped anyway.

These assert on presence, not on readiness, so they survive whatever partial-group recovery ends up looking like. recovered counts a group if the restored buffer knows about it at all — already committed with its missing rollouts regenerated, or a reserved slot for the run to finish. Either design flips these to XPASS, and strict=True makes that a failure that has to be dealt with.

How to read the matrix

Two plain tables, so the setups can be scanned without the machinery:

# test_checkpoint_regressions.py
REGRESSED = [Case(s, sampler) for sampler in GATED_SAMPLERS for s in ALL_SCENARIOS]

# test_checkpoint_no_data_loss_xfail.py
EXPECTED_TO_FAIL = [
    Case(S_PARTIAL,              "windowed", IN_FLIGHT),
    Case(S_LAG2,                 "windowed", IN_FLIGHT),
    Case(S_EVICTED,              "windowed", IN_FLIGHT),
    Case(S_TRAINED_OUT_OF_ORDER, "windowed", IN_FLIGHT),
]

A scenario reads the way you would say it out loud:

S_PARTIAL = Scenario(
    name="lag1-next-step-partly-generated",
    groups=(
        Group(9, 2, weight=0), Group(10, 2, weight=0), Group(11, 2, weight=0),
        Group(12, 1, weight=1, target=5),   # one rollout still running
        Group(13, 2, weight=1, target=5),
        Group(14, 0, weight=1, target=5),   # not started
    ),
    cursor=15,
    trained=frozenset({9, 10, 11}),
    lag=1,
)

Fidelity

Real TQReplayBuffer, real reserve/commit, real metadata_state_dict/load_state_dict, real manifest digest, real samplers via create_sampler, real NoOpDataPlaneClient.save_checkpoint/load_checkpoint. The save/restore gate mirrors the two call sites above by reading the sampler's own supports_buffer_checkpoint. Only the tensor converter is stubbed, so a scenario can use empty prompt records.

Testing

cd tests && uv run pytest unit/single_controller/test_checkpoint_no_data_loss.py \
                          unit/single_controller/test_checkpoint_regressions.py \
                          unit/single_controller/test_checkpoint_no_data_loss_xfail.py -v

No GPU, no Ray cluster, no TransferQueue process. Collected by the existing L0_Unit_Tests_Other lane. Note the repo's addopts includes -x, so a CI run stops at the first regression rather than reporting all 10.

Open questions

1. Is losing the gated-sampler restore a deliberate trade? If one durable store instead of two is worth it, the right resolution is not to xfail these but to decide the old behaviour is not coming back and say so where someone configuring in_order will read it. If it is not deliberate, the tensors are still in TransferQueue and the fix is index-only.
2. Does feat(sc): gate-authoritative token capture on the Gym tokidcap stack #3456 lead to partial-group recovery? It stages token lineage for running generations, which looks like the prerequisite, but nothing states that a restore would then finish a half-done group. If that is out of scope, the 4 xfail rows may belong to a later change.
3. Does S_EVICTED model eviction the way you would? An evicted group is treated as an intentional discard and excluded from what a restore must return.

Happy to drop, re-scope, or re-target any of this — the tables are meant to be argued with.